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A Frame Rate Optimization Framework For 
Improving Continuity In Video Streaming 

Evan Tan and Chun Tung Chou 
Abstract 

This paper aims to reduce the prebuffering requirements, while maintaining continuity, for video streaming. 
Current approaches do this by making use of adaptive media playout (AMP) to reduce the playout rate. However, 
this introduces playout distortion to the viewers and increases the viewing latency. We approach this by proposing 
a frame rate optimization framework that adjusts both the encoder frame generation rate and the decoder playout 
frame rate. Firstly, we model this problem as the joint adjustment of the encoder frame generation interval and the 
decoder playout frame interval. This model is used with a discontinuity penalty virtual buffer to track the accumulated 
difference between the receiving frame interval and the playout frame interval. We then apply Lyapunov optimization 
to the model to systematically derive a pair of decoupled optimization policies. We show that the occupancy of the 
discontinuity penalty virtual buffer is correlated to the video discontinuity and that this framework produces a very 
low playout distortion in addition to a significant reduction in the prebuffering requirements compared to existing 
approaches. Secondly, we introduced a delay constraint into the framework by using a delay accumulator virtual 
buffer Simulation results show that the the delay constrained framework provides a superior tradeoff between the 
video quality and the delay introduced compared to the existing approach. Finally, we analyzed the impact of delayed 
feedback between the receiver and the sender on the optimization policies. We show that the delayed feedbacks 
have a minimal impact on the optimization policies. 

I. Introduction 

Video continuity is the length of time the video is being played without interruptions. A low video continuity 
would result in a stop-start video that is known to impact on the viewer perceived video quality |jT|. One of the 
main causes of video discontinuity in streaming video is due to buffer underflow caused by for example network 
delay. The traditional way to reduce the buffer underflow occurrences is to prebuffer video at the decoder. The 
prebuffering amount needs to be sufficiently large to maintain video continuity. However, a large prebuffering 
introduces unwanted initial delays into the system. 

To reduce the amount of prebuffering required while maintaining video continuity, current approaches make use 
of adaptive media playout (AMP) |l2l, S, H. AMP approaches reduce the playout frame rate of the decoder 
in order to avoid buffer underflows, this is because slowing down the playout frame rate is preferable to halting 
the playout [5]. The playout frame rate is defined as the rate of frames being removed from the decoder buffer 
for decoding and playout to the viewer. AMP has been shown to reduce prebuffering while maintaining video 
continuity |3|. However, it introduces playout distortion to the viewers, this is because the video is being played 
slower than its natural playout frame rate (also known as the video capture frame rate). Furthermore, AMP schemes 
adjust the playout frame rate independent of the encoder strategy, this potentially introduces more playout distortion 
than required. While the playout distortion can be reduced by limiting the playout slowdown, it potentially affects 
the video continuity. Moreover, slowing down the playout frame rate increases the viewing latency and introduces 
additional delay into streaming system. 

In this paper, we aim to reduce the amount of prebuffering required by dealing with the continuity of the video 
in a reactive manner. We realize this aim by proposing a framework that performs frame rate control at both the 
encoder and decoder as well as introduce a way to constrain the viewing latency. The key idea is that, if the network 
bandwidth drops and the number of frames in the decoder buffer is low, the encoder should send more frames to 
the decoder to prevent buffer underflow from occurring, thus maintaining playout continuity. In order to approach 
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this systematically, we formulate an optimization problem that takes into account the video continuity, video quaUty 
and overall play out delay. The contributions of this paper are: 

1) We propose a frame rate control framework that jointly adjusts the encoder frame generation rate and the 

playout frame rate. This distinguishes our framework from conventional approaches that do not perform frame 

rate control and AMP approaches that only adjusts the playout frame rate. 

We derive an optimization formulation of the frame rate control framework using the technique of virtual 
buffer. We then use Lyapunov optimization |6| on this model to systematically derive the optimization policies. 
We show that these policies can be decoupled into separate encoder and decoder optimization policies with 
feedback between the two. This allows for a distributed implementation of the policies. We demonstrate that 
this framework produces a very low playout distortion in addition to a significant reduction in the prebuffering 
requirements compared to existing approaches. 

A delay constraint that reduces the accumulated delay from playout slowdowns. We then show that the 
delay constrained framework provides a superior tradeoff between the video quality and the delay introduced 
compared to the existing approach. 

An analysis of the impact of delayed feedback between the receiver and the sender. We show that the delayed 
feedbacks have a minimal impact on the optimization policies. 
This paper is organized as follows. Section [IT] reviews the current approaches. Section IV demonstrates how the 
frame rate control problem can be modelled. Section |V] demonstrates how a delay constraint can be introduced 
into the framework. Section VI analyzes the delayed feedback on the optimization policies. Section VII describes 
the video quality functions that can be used with this framework. Section VIII evaluates the performance of the 
proposed improved framework. Section llX] concludes this paper. 
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II. Related Work 

Prebuffering video data has been studied in |T|, fW\, f9\. These techniques are focused on calculating the con^ect 
amount of prebuffered data based on a mathematical model to avoid the occurrence of a buffer underflow once the 
video playout has started. However, these techniques do not take into account of the varying playout rates and the 
resulting reduction in prebuffering into their models. Furthermore, prebuffering introduces unwanted delay into the 
system and is known to have an impact on the user perceived quality of the video [10|. 

Sender rate adaptation techniques such as encoder rate control lITTll . |[12| and scalable rate control |[T3l achieve 
video continuity by ensuring the video rate matches the network bandwidth. However, these approaches do not 
take into account of the network delay variation that might occur (e.g. due to network jitter). Moreover, it has 
been demonstrated that coupling AMP with sender rate adaptation techniques can further reduce the prebuffering 
requirements (Si- 

AMP is another class of techniques that is used to improve video continuity. The main idea behind AMP is to 
reduce the playout rate of the received media. This reduces the departure rate of the media in the decoder buffer 
and potentially allows the buffer to be filled up to safer levels. 

AMP has been studied extensively for audio applications, (Ml, IBl, Ull, El, HH, HSl. These audio AMP 
techniques depend on audio scaling techniques such as WSOLA EOll . which allows audio to be scaled without 
changing its pitch. A recent work by Damnjanovic et al has shown that audio and video scaling can be done in 
real-time [21J. In this paper, we focus on AMP for video. 

AMP for video involves scaling the frame intervals to slowdown or speedup the playout rate. The slowdown 
or speedup is triggered by a threshold set on the buffer occupancy. Once the buffer occupancy drops below the 
threshold, AMP will slow down the playout rate and vice versa. There have been studies conducted on the dynamic 
adjustment of this threshold 13, |[22.1 . 1231 . [24] . The adjustment is normally based on the network condition, the 
poorer the network condition, the higher the threshold. These techniques mainly based the threshold on the buffer 
occupancy. We instead record the difference between the receiving frame interval and the playout frame interval 
into a virtual buffer and treat it as a penalty, which is equivalent to soft thresholding. 

AMP has also been integrated into the design of packet schedulers |[25l . ll26l . ll27l . |[28i . ||29]| . These techniques 
tend to slowdown the playout rate for important video packets. This ensures that the more important video packets 
have a higher probability of meeting its deadline and thus avoid being dropped. We do not focus on packet scheduling 
in this paper and our proposed framework could complement any packet scheduling scheme. 



3 



Another aspect of AMP being studied is the smoothness of transition between playout rate adjustment ||4l, Il30ll . 
||3T1 . The goal of these approaches is to ensure that adjustments made to the playout rate is done as smoothly as 
possible so as to reduce any noticeable effects to the viewers. We do not focus on rate smoothing in the current 
paper and again any smoothing scheme can be used within the framework. 

Steinbach et al ||32|| and Kalman et al |l3l both examined the trade-off between delay and buffer underflow using 
a two state Markov models. Kalman et al further proposed AMP-Initial, AMP-Robust and AMP-Live. AMP-Initial 
slows down the playout rate until the buffer reaches a certain target level, this produces a lower perceived initial 
buffering delay to the viewer. AMP-Robust slowdowns the playout rate if the current buffer occupancy falls below 
the target buffer level while AMP-Live slowdowns or speedups the playout rate to maintain the target buffer level. 

These AMP approaches mainly examine only the effects of adjusting the playout frame rate independently and 
do not consider any encoder strategy to reduce playout distortion. While our proposed approach aims to examine 
the effects on video quality and viewing delay based on the adjustment of both the encoder frame generation rate 
and the playout frame rate. Slowing down the playout frame rate also introduces viewing latency, we will propose 
a way to constrain this latency in our proposed approach. 



III. Frame Rate Control For Video Continuity 
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Figure 1. Encoding-decoding flow with encoder frame generation rate i{t), sending frame rate fj,{t), receiving frame rate X{t) and playout 
frame rate o{t). All rates are in frames per seconds. 



Figure [T] shows a typical video transmission scenario. The encoder generates video frames at a rate of i{t) frames 
per second (fps) at time t into the encoder buffer. The network transport protocol then transmits the video data 
from the encoder buffer into the network at a rate of fps. The transmitted video data will be received at the 
decoder at a rate of X{t) fps in its buffer. The decoder then proceeds to playout the received video data from the 
decoder buffer at a rate of o{t) fps. Video data arriving after the playout deadline are assumed to be lost. In the 
scenario described in fig. [TJ there are two mechanisms that can be used to maintain video continuity: reducing 
playout frame rate o{t) and increasing the encoder frame generation rate i{t). 

Our analytical model takes network delay into consideration but does not consider packet losses In particular, 
an important factor that affects the performance is the inter-frame delay, which is defined as the difference of 
network delays for consecutive video frames. We assume that the frames are received into the decoder buffer in 
display order, so reordering is done before the frames enter the decoder buffer. Thus, inter-frame delays can be 
affected by network jitter and packet losses. Notice that this model allows us to focus on calculating the amount 
of frames arriving into and departing from the decoder buffer such that decoder buffer underflow can be avoided, 
and video continuity can be maintained. We also assume, in this paper, that the encoder is able to generate frames 
faster than the natural playout frame rate. This is a reasonable assumption since there are existing encoders such 
as x264 [33 1 that can encode frames in real-time faster than a typical natural playout frame rate of 30 frames per 
second (fps). 

Slowing down the video playout frame rate o{t) is termed in the literature as AMP. The idea is that the slower 
video playout allows time for the buffer to fill up to the required level without the need to stop the video for 
rebuff ering. 

Another way to maintain video continuity is to increase the encoder frame generation rate i{t), which refers 
to the amount of frames the encoder actually sends out into the network and it is important to point out that 
the encoder frame generation rate is not a temporally scaled frame rate. Note that we purposely chose the term 
encoder frame generation rate to differentiate it from the commonly used term of encoder frame rate because they 
represent two different concepts. The meaning of encoder frame generation rate is best illustrated by an example: 
when no frame rate control is used, the encoder frame generation rate i{t) is the natural playout frame rate. Let us 



'Note that even though we do not consider packet loss explicitly, the simulation results in section VIII show that our optimisation framework 
works well in the presence of packet loss. 



4 



assume that to be 30 fps; when frame rate control is used, the encoder may increase the encoder frame generation 
rate i{t) to say 60 fps to quickly fill up the decoder buffer to maintain continuity. Note that these 60 frames are 
still encoded using the same natural playout frame rate of 30 fps, so they form the next 2 seconds of video. This 
example illustrates that a higher encoder frame generation rate means more than one second of video is generated 
in one second but the video is always encoded using the same natural playout frame rate. 

Increasing i{t) allows potentially more frames to reach the decoder and increase the decoder buffer level. This, 
in turn, helps to improve the continuity of the video. However, increasing i{t) is likely to cause the video bitrate to 
increase as more frames are produced per second by the encoder. To ensure that the video bitrate does not exceed 
the available bandwidth, we introduce additional compression to the video such that the higher the encoder frame 
generation rate i{t), the higher the compression applied. 

For a given encoder generation rate i{t), the higher compression is obtained by the rate controller adjusting the 
encoder such that the average frame size produced by the encoder is: 

ABR{t) 

average frame size = r{i[t)) = — — (1) 

where ABR{t) is the available bandwidtlj^ at time t. In practice, the average frame size produced by a rate 
control strategy may not satisfy ([T]). However, ([T]) is what most rate control schemes try to meet as they use ([T]) as 
part of their fluid flow model to determine the amount of bits to allocate to a frame ifTTl . l[T2l . l[34l . 

It can be shown that when ([T]l is satisfied by the rate controller, the resulting video bitrate will not exceed the 
available bandwidth for different values of the encoder frame generation rate i{t). Also, the average frame size 
decreases when i{t) is increased which would tend to reduce the frame quality. Finally, a higher i{t) also allows 
more frames to be sent to the decoder, thus it allows us to increase the buffer level and maintain video continuity. 

Reducing o{t) and increasing i{t) increases playout distortion and reduces frame quality respectively while 
improving video continuity. This suggests that an optimal trade-off need to be found. To do this, we model the 
frame rate control problem as an optimization problem and make use of Lyapunov optimization to obtain policies 
that help determine the optimal trade-off. 

In this paper, we adjust the encoder frame generation rate i{t) and playout frame rate o{t) by adjusting the 
encoder frame generation interval ^ and the playout frame interval ^ respectively. The reason we chose to 
adjust the intervals is because it allows the optimization problem to be concave and, therefore, easier to solve. This 
issue will be discussed in more detail in Section IVIH 



IV. Discontinuity Penalty For Frame Rate Optimization 

A. Buffering Criteria 



Figure 2. Receiver buffer model. 



Since the video discontinuity is correlated to decoder buffer underflows, we first study how buffer underflow 
occurs. To do this, we make use of the receiver model illustrated in fig. [2] We assume that the sender, network 
and receiver all work in slotted time. At time slot t, the receiver receives A(t) frames (receiving frame rate) and 
stores it into the receiver buffer. Simultaneously, o{t) frames are being removed from the buffer and played out to 
the viewer (playout rate) at time t. Let b{t) be the amount of buffered video frames in the buffer at time t and T 
be the length of the sequence in time slots, then to avoid an underflow the following condition needs to be met: 

T T 

h{t)>Y,o{T)-Y,Kr) yT>t (2) 

T=t T=t 

^ideally the encoder should use ABR{t + rfs), where ds is the sender buffer delay. However, since this is difficult to determine, we use 
ABR{t) to approximate ABR{t + d^). 
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Equation Q intuitively means that, to avoid a buffer underflow, the cumulative buffer drainage for the rest of 
the sequence playing time should not exceed the current buffer occupancy b{t). 

Now we refine the above model to make use of frame intervals. This is done to build up a system model using 
frame intervals. Basically to make use of the frame intervals, we set \{t) = ^ and o{t) = Equation Q then 
becomes: 



^ r(r) -p{t) 



b{t) > ^ \' (3) 

We assume that < T{t) < Vmax and Pmin < p{t) < Pmax, i-e. both r{t) and p{t) are bounded. That would 
mean that we can approximate ([3]) as: 

T 

bit) Tmin Pmin > ^ ^'(t) - p{r) (4) 

T=t 

Note that the choice of using rmin and Pmin results in a more conservative bound. An alternative bound, which 



is looser, is to replace the left-hand-side of Q by b{t) r^axPrnax- We will show in simulation results (section Villi 
that these two choices give similar results. 

If we divide Q by T — t, the remaining time slots left, we can estimate the buffer underflow bound per time 
slot. This will result in: 

— >r — p (5) 

T-t - ^ 

where f and p are the averages of r{t) and p{t) respectively. Equation ([5]) provides us with a way to design the 
optimization policy to avoid buffer underflow in a time slot. Since Lyapunov optimization works on a per time slot 
basis, we prefer ([5]) over Q. Essentially, we need to design a policy that produces a receiving frame interval r{t) 
and a playout frame interval p{t) in such a way that the above bound is met. 



B. System Model 
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Figure 3. System model showing the frame intervals. 



We now show how r{t) and p{t) are produced in the complete system model. Fig. |3] illustrates the system model. 
In a time slot t, the encoder at the sender produces video frames at intervals of f{t) and stores them into the sender 
buffer. The sender sends the video frames in its buffer at intervals of s{t). Note that f{t) and s{t) are the encoder 
frame generation interval and the sending frame interval respectively. Simultaneously, the receiver receives video 
frames from the sender at intervals of r{t) and puts them into the decoder buffer. The decoder in the receiver plays 
out the video frames at intervals of p{t) to the viewer, p{t) is the playout frame interval. The network will also 
produce a forward delay of and a backward delay of df,. 

The goal of our framework is to jointly adjust the encoder frame generation interval f{t) and the playout frame 
interval p{t) to maintain video continuity. To simplify the model, we assume that f{t) = s{t), while this essentially 
assumes no delay caused by the sender buffer, delays caused by the sender buffer is simulated in our experiments 
later on. We also assume that f{t) is bounded within the range [fmin, fmax] and r{t) is defined by a network delay 
variation function F{s{t)) as : r{t) = F{s{t)) = F{f{t)), since f{t) = s{t). 

This means that we can represent the delay variation based on the encoder frame generation interval f{t). In 
this paper, we specify F{f{t)) as: 
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F{f{t)) = e{t) X fit) (6) 

Where e{t) is the frame interval scaUng factor due to delay variations from the network. In practice, we estimate 
e{t) at the receiver by: 

Where dj is the forward delay between the sender and receiver. Note that if there is no delay (i.e. db = dj = 0), 
it will mean that F{f{t)) = r{t). 

C. General Optimization Problem 

There are three main objectives that we want to optimize: 1. frame quality, 2. playout distortion and 3. continuity. 
Frame quality is defined as the perceived visual quality of the video. This will be represented as a frame quality 
function g{f{t)), where g{f{t)) is an increasing function of f{t). Playout distortion is the perceived distortion 
when the playout rate deviates from the natural frame rate and will be represented by a function h{p{t)), where 
h{p{t)) is a convex function of p{t). Both g{f{t)) and h{p{t)) are non-negative functions, i.e. g{f{t)) > and 
h{p{t)) > 0, and are assumed to be uncorrected. We will suggest a specific form for g{f{t)) and h{p{t)) later 



in Section VII Continuity is the length of time the video is played without interruptions due to buffer underflow. 
We ensure continuity in this framework by ensuring that a virtual buffer stabilizes, this concept will be explained 
further in the next section. 

With the system model described in the previous section, we now formulate a general optimization problem: 



Maximize: g{f{t)) - h{p{t)) (8) 
Subject to: U{t) is stable (9) 

fmin <fit)< fmax (10) 
Pmin < P{t) < Pmax (11) 

Where U{t) is the virtual buffer representing the discontinuity penalty. Since the objective ([8]) is separable, 
maximizing ^ can be seen as maximizing the frame quality function g{f{t)) and minimizing the playout distortion 
function h{p{t)). The constraint ([9]) is the continuity constraint. Constraints ([10]) and (VV) are the limits set on f{t) 
and p{t) respectively. 

To see how the above general optimization problem is derived, we first replace (|9]) with a continuity constraint 
derived from Q: 

E{F(/(t))-p(t)} </3(t) (12) 

where /3(t) = ^(*) '''■^i^P^ai^ _ Constraint ( [T2| ) can be satisfied by either decreasing f{t) and/or increasing p{t). 
However, decreasing f{t) would result in a lower frame quality given by g{f{t)) and increasing p{t) would result 
in a higher playout distortion given by h{p{t)). The optimization policy would need to handle these tradeoffs. To 
solve this optimization problem, we make use of the concepts of virtual buffer and Lyapunov optimization. 

D. Virtual Buffer And Stability 

Virtual buffers are a concept introduced by Neely et al |[35l to replace certain constraints of an optimization 
problem. To determine a suitable virtual buffer for our problem, we make an initial virtual buffer design to represent 
the continuity of the video. The virtual buffer will be updated at every time slot t, and is updated as: 

U{t + l) = [U{t)-p{t)\+ + F{f{t)) (13) 

where f/(0) = 0. The virtual buffer U{t) is lower bounded by (i.e. always non-negative). This virtual buffer can 
be seen as the discontinuity penalty. If F{f{t)) is higher than p{t), it means that the network throughput is lower 
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than the playout rate, i.e. the rate of video frames received is slower than the amount of video frames being played 
out. Thus, the discontinuity penalty U{t) accumulates the difference of F{f{t)) —p{t) as a penalty. If the network 
subsequently improves and F{f{t)) is lower than p{t), the penalty in the buffer then reduces by p{t) — F{f{t)). The 
higher U{t) becomes, the higher the possibility of a buffer underflow. In an ideal network, where F{f{t)) < p{t) 
at all times, U{t) will never accumulate any penalty. 

To ensure that U (t) does not keep increasing, we want U (t) to stabilize. We will show later that by stabilizing 
U{t), we can ensure that the video continuity is preserved. The definition of stability used here is E{C/} = 
limsup(_^o^ J St=o^i^(^)} ^ "^^^^ intuitively means that the buffer is stable if it does not grow infinitely 



large over time. We also like the virtual buffer to meet the continuity constraint ( [T2| ) when it stabilizes. To do that, 
we extend the initial virtual buffer ( [T3| ) as: 

U{t + 1) = [U{t) - Pit) - /3(t)]+ + (14) 

To see how it works, notice that for the discontinuity penalty U{t) to grow infinitely, the following condition 
needs to be met: F{f{t)) — p{t) > P{t). Therefore, in order for U{t) to stabilize the following condition needs to 



be true: F{f{t)) — p{t) < P{t). This is the continuity constraint as defined in ([12]), so when U{t) stabilizes, the 
continuity constraint will be met. The discontinuity penalty U (t) is maintained at the receiver in our design. 
Thus, with the stability of the discontinuity penalty U{t) as a constraint, we then obtain the general optimization 



problem presented in Section IV-C We will demonstrate in Section VIII-C using simulations, that U{t) is positively 



correlated with the video discontinuity. 

E. Lyapunov Optimization Derivation 

We show here how we convert the optimization problem presented in Section IV-D into a separate encoder and 
decoder optimization policies using Lyapunov optimization. We assume that there is no network delay between the 
sender and receiver (i.e. dj = = 0). This is to simplify the analysis presented here. We will relax this assumption 
in Section |Vll 

We define a Lyapunov function L{U{t)) to represent the "energy" of the discontinuity penalty U{t) at time t, 
this can be any arbitrary non-negative function. We use the following Lyapunov function in this paper: 

L{U{t)) ^ ^ (15) 
We then define the one-step conditional Lyapunov drift A{U{t)) as: 

A([/(t)) ^ E{L{U{t + 1)) - LiumU{t)} (16) 



Equation ( [T6| ) can be understood as the expected change in the energy in one time slot step. The goal of Lyapunov 
optimization is to show that this energy reduces or stays the same at each slot (i.e. ([T6]l produces a negative or zero 
drift). This would ensure the stability of the buffer, which in turn enforces the continuity of the video playout. To 



show that U{t) stabilizes, we need to convert the buffer update equation ( [141 ) ii^to ^ one-step drift ( [16] ). 

To do that, we square the buffer update equation ([14]), divide it by two, and take expectations of the result (see 
Section [a] in the appendix for details), we will get the following expression: 

A{U{t)) <B- U{t)E{P{t) +p{t) - F{fmU{t)} (17) 
Where i? is a constant defined as: 

D ^ f 2 , ( , '^min Pmin\ \ ^,o\ 
= - r^ax + Pmax + 7^ : (18) 



2 \ max ' 1 I rp ^ 

From (17), we can use the results in |[6l to prove that U{t) stabilizes (see Section [d] in the appendix). Once 
U{t) is proven to stabilize, it can be shown using the results in f35| that the continuity constraint ( [T2] ) is satisfied. 
However, to optimize the frame quality utility g{f{t)) and the playout distortion h{p{t)), we need to massage the 
equation more. By subtracting from both sides, the term V¥,{g{f{t)) — h{p{t))\U {t)} , which is the expectation of 
([8]) scaled by a positive constant y > 0, and by rearranging the terms. We get: 
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A{U{t))-VE{g{fit))-h{pmUit)} 
<B-E{Uit)m\Um 

-E{Vg{f{t))-U{t)F{fmum 

-E{Uit)pit)-VhipmUit)} (19) 



The third term on the right hand side of ( [T9| ) is a function of the encoder frame generation interval f{t) only 
and represents the sender optimization policy. The last term of ([19]) is a function of the playout frame interval p{t) 
and represents the receiver optimization policy. 

To summarize: 



1) Encoder optimization policy: From the third term of (^9), based on the frame interval scaling factor e(t) 
(defined in ([7])) and discontinuity penalty U (t) feedback from the receiver. The encoder in the sender will calculate 
F{f{t)) using ([6]), and it will choose f{t) at each time slot as the solution of the following optimization: 



Maximize: Vg{fit)) - 

Subject to: /™„ < f{t) < f^ax (20) 

2) Decoder optimization policy : From the last term of ( [T9] ), the decoder in the receiver will observe the current 
discontinuity penalty U (t) and choose p{t) at each time slot as the solution of the following optimization: 



Maximize: U{t)p{t) - Vh{p{t)) 

Subject to: pmin < p{t) < Pmax (21) 

Notice that the optimization policies are decoupled into separate optimization subproblems for playout frame and 
encoder frame generation intervals. Note that the decoder is responsible for updating U {t) using equation ( [T3] ) and 
sending the value of U (t) to the encoder. Furthermore, under appropriate conditions, the decoupled problems are 
convex. This makes the problem easier and more flexible to solve. Given that Lyapunov optimization minimizes 
the right hand side of ([T9j) instead of the original optimization problem ([8]) (which is hard to solve), the objective 
function value realized by Laypunov optimization is sub-optimal but its deviation from the optimal value can be 
controlled (see Section |D] in the appendix). 

V. Delay Constrained Frame Rate Optimization 

While slowing down the video playout allows us to reduce the occurrences of buffer underflows and preserve 
the video continuity, it introduces extra viewing latency to the viewers. This becomes an issue when the system 
has a delay budget, as frequent playout slowdowns might cause the delay budget to be exceeded. 

In this section, we examine the problem when there is a constraint imposed on the viewing latency. We focus 
on constraining the additional playout latency generated by slowing down the playout. More specifically, we want 
to impose a constraint on how often playout slowdowns occur and how much the playout can be slowed down. 
This is done by introducing another virtual buffer, called the delay accumulator, into the problem to represent the 
constraint on the accumulated delay due to playout slowdowns. We then use Lyapunov optimization to ensure that 
this constraint is met. 



A. Delay Constrained Policy Design 

The delay constraint can be described as constraining the accumulated playout slowdowns used over the lifetime 
of the whole video sequence. Specifically, let 9 be the maximum playout slowdown delay tolerable for the video 
application and pn represent the natural playout interval of the video. Then, the constraint could be written as: 



T-l 

T=0 



(22) 
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Recall that T is the length of the sequence in time slots. Since Lyapunov optimization works on a per time slot 
basis, we need to express the constraint (|22]) as a constraint for each time slot. To do this, note that for the current 
timeslot t, the averaged maximum playout slowdown delay tolerable will be: td = tJ-t, where T — t represents the 



number of time slots remaining for the sequence. With this, we can rewrite the delay constraint (22 1 as: 



EMt) -Pn}<td 

Then, what remains to be done is to design a policy that ensures that constraint 



(23) 

is met at each time slot. 



It can be seen that by adding constraint p3] ) to the general optimization problem presented in Section IV-D 
obtain a problem that provides video continuity while ensuring that the delay constraint is met. 



we 



B. Delay Accumulator Virtual Buffer 

To apply the delay constraint ([23]) into the framework using Lyapunov optimization, we again make use of the 
virtual buffer concept. We introduce another virtual buffer named as the delay accumulator. The delay accumulator 
is a virtual buffer that keeps track of the accumulated delay caused by playout slowdowns. Everytime a playout 
slowdown is perform, the delay accumulator will increase correspondingly. However, the delay accumulator will 
reduce when playout speed up is perform. Formally, the buffer dynamics to describe the delay accumulator for 
each time slot would be: 



X{t + l) = [Xit)-pn-td]-^ +pit) (24) 
Note that the terms p„, p{t) and are taken from constraint ( |23l ) above. Therefore, by showing that the delay 



accumulator X{t) stabilizes, we can show that the delay constraint (23 1 can be satisfied |[35i . This also means that 
the delay constraint can be written as: X{t) is stable. We now show how Lyapunov optimization can be used to 
derive optimization policies to solve the above problem. 



C. Lyapunov Optimization Derivation 

Note that there are two buffers in the problem, the discontinuity penalty U{t) and the delay accumulator X{t). 
To stabilize both of these simultaneously, we first redefine the Lyapunov function to be: 

The one step conditional drift also needs to consider both buffers and is defined as: 



A{U{t), X{t)) = E{L{U{t + l),X{t + 1)) 

-L{U{t),XmU{t),X{t)} (26) 

To shorten the formulas, we use A, U, X, p and / to represent A{U{t), X{t)), U{t), X{t), p{t) and f{t) 
respectively. By squaring (14i and (24i, taking expectations and dividing by 2, we get (see Section |B] in the 
appendix for details): 



A{U,X)<B + C - C/E{p -F{f)\ U, X} 
-XE{pn + td-p\U,X} 



(27) 



where B is defined as in (1 8 1 and C 



Pmax + iPn + td) 



It can be proven that ( [27] ) results in stability for both the discontinuity U and the delay accumulator X (35] (see 
Section [E| of the supplementary materials). Furthermore, it can be proven that the stabilization of X implies that 
the constraint ( [23) can be met by using the results from |[35l . 

To optimize the frame quality and the playout distortion, we subtract V¥,{g{f{t)) — h{p{t))\U {t)} from both 



sides of (27) to obtain: 
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A{U,X)-VE{g{f)-h{p)\U,X} 
<B + C-XE{pn + td\U,X} 

-E{Vg{f)-UF{f)\U,X} 
-E{Up- Vh{p) - Xp\ U, X} 



(28) 



Note that the encoder optimization policy (second last term) remains the same as ( |20| ). The last term shows that 
decoder optimization policy ( |2T] ) has an additional penalty term —Xp. This will mean that as X increases, the 
decoder will get penalized more for high playout interval p values. This will encourage the decoder to choose a 
lower p whenever the accumulated delay in X is high. Lastly, the performance bound for the Lyapunov optimization 
can be derived (see Section |E] of the appendix). 

VI. Network Delay Impact 

In Sections IV and[V] we showed how Lyapunov optimization can derive optimization policies that help ensure the 
video continuity and enforce a delay constraint. However, the assumption made in those sections for the derivations 
is that there is no network delay. Specifically, we assumed that the feedback delay from the receiver to the sender 
is non-existent. In this section, we relax this network delay assumption and analyze the impact of network delay 
on the optimization policies. 

Recall that the network generates a forward delay of and a backward delay of from the system model 
discussed in Section IV-B With network delay, the discontinuity penalty U {t) updating in the receiver is performed 



as: 



U{t + l) = [U{t)-^{t)]+ + F{f{t-dj)) 



(29) 



where ^{t) = p{t) + /3(t) and recall that (5{t) 



bjt) r„ 



T-t 



'—. Note that the difference between the previously 



presented discontinuity penalty buffer dynamics in ( [T3] ) and above is that ( |29l ) is based on the forward delayed 
encoder frame generation interval f{t — df). Furthermore, the sender relies on feedback from the receiver, so it 
chooses f{t) based on a backward delayed U{t — db). There are two possible issues that might impact on the 
optimization policies when delays are present: 

1) the delayed discontinuity penalty U{t) feedback from the receiver to the sender means the encoder optimization 
policy will need to make use of U{t — db). 

2) the current choice of f{t) at the sender would only affect receiver dj time slots later. 

What we need to do is to derive optimization policies that take the above two issues into account and show that 
these policies stabilizes the discontinuity penalty U{t). 
To begin, note that pQ]) can be represented recursively as: 



Uit + l) = [U{t)-jit)]+ + Fif{t-df)) 

U{t) = [U{t-l)-^{t-l)]+ + F{f{t-df 



1)) 



Uit -di, + l) = Pit - db) - 7(i - db)]+ + F(/(t -df- db)) 
This implies that U{t) can be recursively defined as: 



U{t)< 



t-i 



u{t-db)- ^(^) 



=t-db 



t-df-l 



(30) 



(31) 



-t—dh—df 



Issue 2 suggests that we need to predict dr time slots in the future. To do that, we change the buffer updating 



equation ( [29| ) into a. df slot update: 
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U{t + df + l) < 



t+df 



U{t) - 



T=t 



+ 



E F{f{r)) 



(32) 



=t-df 



What ( [32] ) means is that the receiver updates the discontinuity penalty U (t) not only based on the known current 
values 7(t) and f{t — df) but also using the predicted values dj slots in the future, [y{t + 1) . . . j{t + df)] and 
[f{t — df + 1) . . . f{t)]. Note that F(/(r)) in ( [32] ) is calculated from df time steps in the past. This is because the 
current f{t) only affects the discontinuity penalty df time slots later which will be U{t + df + 1). The df step 
buffer dynamics can be proved to obtain stability by using the T-slot Lyapunov drift 



To show how the T-slot Lyapunov drift achieves stability, we first convert the 1-step Lyapunov drift in ( [T6| ) into 
a dj-step drift: 



A{U{t)) ^ E{L{U{t + df + 1)) - L{U{t))\U{t)} 

We now use the following shortened notations to simplify the equations: U = U{t), Ud^ = U{t 

Et=*-.. 7(r), 7d, ^ Ettt 7(r), F ^ T!;Xl-d, nf{r)) and F^, ^ ZUt-d, ^(/(r)} 

Using the same Lyapunov function ( fTS] ) as in the previous section and drift definition 
divide it by two and take expectations (see Section [C] of the appendix for details), we get: 



(33) 

db), 7 = 



If we square (32i, 



AiU) < B' -E{U-fa,\U} +E{UFd,\U} 



Where: 



B' 



2 



+ 



Tr. 



mm ymm 

T-t 



(34) 



(35) 



Equation (34i can be shown to achieve stability 11361 . effectively settling issue 2. However, to deal with issue 1, 



we need to show how the feedbacked discontinuity penalty U{t — db) affects (|34]). To do that, we substitute the 
recursively defined U{t) ([3T]) into ([34]): 



A{U) < B' -E{U-/d,\U}+E{{[Ud,--/]-^ + F)Fd,\U} 



(36) 



Recall the definitions of F and F^f, and given that F{.) is an increasing function of f{t) (see Q), this implies 
that F and F^j: can be bounded as F < dbrmax and F^^ < {df + l)rmaa; respectively. It then follows that FF^f 
can be bounded as: 



FFd, < db{df + l)r 



2 

max 



(37) 



Note that [Ud^ -l] <Ud^- Thus, we use ([37]l in (|36]l, we get 



A(C/)< B" -UE{^ds\U] -E{-Ud,Fd,\U} 



(38) 

where B" = B' + db{df + 1) r'^ax with B' from ( [35] ). Equation ( [38] ) in that form can be proven to stabilize by 
using the results from Il36l (see Section [F] in the appendix), thus settling issue 1. 

As in Section IV-E[ to cater for utility optimization, we subtract from both sides, the term VE{g{f{t)) — 
h{p{t))\U} and rearrange the terms to obtain: 



A{U)-VE{g{f{t))-h{pmU} 

< B" -E{Ujd,-Vh{p{t))\U} 

-E{Vg{f{t))-Ud,Fd,\U} (39) 

Using the definitions of jdf and Fd^ and that j{t) = p{t) + f3{t). Equation ([39]) can be rewritten as: 
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AiU)-VE{g{fit))-h{pmU} 

( t+df t+df 

< b"-ue\ j2 pi^) + E ^(^) 



T=t + 1 T=t 

t-1 



u 



u 



[r=t-df 

-E{Up{t)-Vh{pmU} 

-E{Vg{f{t))-Ud,F{fmU} (40) 

It can be seen from (|40]) that the last two terms represent the decoder and encoder optimization policies 
respectively. The decoder policy is not affected by the network delay while the only change to the encoder policy 
is to make use of U{t — dh) fed back from the decoder. Moreover, since the delay accumulator X{t) is updated 
locally within the decoder. This would imply that the delay constrained decoder policy ( [28] ) would not be affected. 
Lastly, the performance bound for the Lyapunov optimization can be derived (see Section |F] of the supplementary 
materials). 

VII. Video Quality Functions 

In the previous sections, we showed how using the discontinuity penalty virtual buffer in the system model 
allows us to express the processes as frame intervals. We then used Lyapunov optimization to derive optimization 
policies that stabilizes the discontinuity penalty virtual buffer and showed that this helps to maintain the video 
continuity. We also demonstrated how to add a delay constraint into a framework by using the delay accumulator 
virtual buffer. By deriving optimization policies that stabilze the delay accumulator virtual buffer, we showed that 
the delay constraint can be met. Finally, we studied the impact of network delay on the optimization policies and 
showed how the optimization policies can be derived with network delay consideration. 

What is lacking thus far is a discussion on the specific choice of frame quality function g{f{t)) and play out 
distortion function h{p{t)). In this section, we shall examine the specific forms for g{f{t)) and h{p{t)). 

A. Frame Quality Function 

We first look at an appropriate frame quality function for g{f{t)), g{f{t)) should ideally be concave so that a 
solution can be easily found for the encoder policy pO| . As mentioned before as f{t) decreases, the encoder frame 
generation rate increases. This means that more compression is needed to meet the available network bandwidth 
and more compression tends to mean that the frame quality will be reduced. One of the ways to measure frame 
quality is to measure the peak signal-to-noise-ratio (PSNR) of the video. 

An and Nguyen ||37l has been shown that PSNR can be represented using a log function of the bitrate. We make 
use of their result and fit PSNR to the average frame size of the sequence: 

PSNR{f{t)) = a log{ABR{t) f{t)) + c (41) 

where a and c are the modeling coefficients, and ABR{t) is the current available network bandwidth. Note that 
we make use of all the available bandwidth to transmit the video frames from the sender. Fig. |4] shows the fitted 
curve. The video sequence used for fitting is a concatenation of football, city, crew and akiyo sequences in that 
order. This is done to ensure that the sequence contains several subsequences of different coding complexity. Notice 



that if we use the encoder frame generation rate i{t) as the input variable instead, (41 1 becomes: 



PSNR{i{t)) = a (^^^J^^ + c (42) 

where i{t) = jj^, the resulting PSNR function would no longer be a concave function of i{t) and would make 
the optimization problem more difficult to solve. This is the reason why we use the encoder frame generation 
interval f{t) instead of encoder frame generation rate i{t). 
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Since f{t) is upper bounded by fmax, we fix the maximum of PSNR{f{t)) to fmax- This is done to make the 
PSNR function differentiable in the interval [fmin, fmax] and make the maximization of it easy to calculate. We 
do this by subtracting -p— from the first derivative of {PSNR'{f{t))y. 



PSNR'ifit)) 



fn 



fit) frr 



Integrating (43 1 will give us the desired g{f{t)): 



g{f{t)) = alog{ABR{t)f{t)) + c 



a fit) 

fmax 



(43) 



(44) 



Note that gifit)) is concave between the range 

[fmin J fmax] (whcrC fmin ^ 0). 





Figure 4. PSNR versus average frame size. 



Figure 5. Playout distortion function. p{t) is in milliseconds, 
m — 1 and Pn = ^ ms. 



B. Playout Distortion Function 

In this section, we choose an appropriate playout distortion function for h{p{t)). We modified the version of 
playout distortion function used in 1.29,1 : 

h{p{t)) = m ■ {pn - pit))'^ (45) 

where m is the motion intensity of the sequence, calculated using the technique in f38l, and p„ is the natural 
playout interval. Fig. |5] shows the playout distortion function h{p{t)). h{p{t)) is convex in the range [pmimPmax]- 
Eqn. ( |45] ) is a combination of the quadratic playout distortion function proposed in [25J, [391 and the motion scaling 
used in ||29l . The idea is that the playout distortion increases as the playout rate deviates from the natural playout 
rate. The playout distortion is also affected by the motion intensity of the sequence. Intuitively, higher motion 
sequences increases playout distortion more as the change in motion is more perceivable when the playout rate 
deviates from the natural playout rate. 

VIII. Performance Evaluation 

A. Experiment Setup 

We made use of ns-2 BOll to simulate a network with time-varying data bandwidths. We implemented our 
framework into a x264 encoder ||33l, an open source multi-threaded H.264/AVC encoder. Our implementation 
in x264 simulates the network transmission using network traces. The decoder buffer evolution is simulated by 
tracking the arrivals of video frames from the simulated network and the removal of video frames from the buffer 
for playout. Everytime a frame is encoded, the framework will make a decision on the encoder frame generation 
rate and the playout frame rate at the simulated decoder. This is to simulate the real-time adjustment of parameters 
as the video is being encoded for transmission. Network traces obtained from ns-2 are used to simulate the network 
within the encoder. The ns-2 traces provide the sending and receiving times as well as the loss status of each 
video packet. Every packet produced by x264 is assigned a sending time, receiving time and a loss status. These 
infomation is used to simulate the sender and receiver buffers at a packet level. The video packets that arrived at 
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the receiver buffer are then used to calculate the amount of frames that the decoder buffer contains. A decoder 
buffer underflow occurs when the frames removed from the decoder buffer exceeds the number of frames in it. 
Note that our framework could easily be adapted to multiple pre-encoded copies of the video. 

For the network simulations, we made use of a dumbell network topology with the bottleneck bandwidth set to 
5 Mbits/s. One pair of nodes in the network simulated video streaming using the TFRC transport protocol BTl . 
To generate background traffic, random webpage sessions were used for the other pairs of nodes. All the random 
sessions go through the bottleneck link. The average packet delay from the encoder to the decoder is 235 ms, while 
the feedback delay from the decoder to encoder is 330 ms. These high delay values enable us to show that the our 
proposed distributed frame rate control algorithm works in presence of delay. 

The encoder receives feedback from the decoder and solves the optimization problem to determine the encoder 
frame generation interval f{t). The x264 encoder makes use of this encoder frame generation interval f{t) by 
setting the target frame rate of the rate controller to encoder frame generation rate j^y. The rate controller's target 
frame rate is used to compute the quantization level for each frame. The lower the target frame rate, the lower the 
quantization level and, as a result, bigger frame sizes. More details on x264's rate control can be found in (TT\. 

The test sequence used is a concatenation of football, city, crew and akiyo sequences in that order. This is to 
ensure a mix of high and low motion within the sequence. A 16 minute test sequence is obtained by repeating the 
concatenated sequence. 



The model coefficient a from ( [44| ) is found by curve fitting to be 4.91. The constant V is set to 1. In our 
experiments, we tested the continuity of the video, this is defined here as the amount of time spent playing video, 
specifically: 



Playout Continuity = 1 



rebuffering time 



(46) 



total sequence time 

where the rebuffering time is the time needed to refill the buffer to a certain threshold after an underflow event, 
this is a typical behaviour of a video player [42 1. The rebuffering threshold is set to half of the prebuffer amount. 
The prebuffer amount is varied for each run to determine the performance of each scheme. At the end of each run. 



we will calculate the playout continuity using ( 46 1 for each scheme and make a comparison. 



B. Discontinuity Penalty Lyapunov Optimization Results 
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Figure 6. Correlation between average discontinuity penalty 
and continuity. 
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Figure 7. Prebuffering delay vs continuity. 
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Figure 8. Prebuffering delay vs playout distortion. Figure 9. PSNR loss for LOpt. 
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We first examine tlie (negative) correlation between the discontinuity penalty U{t) and the continuity of the 
video ( |46l ), fig. [6] plots this. It can be seen from the graph that as the discontinuity penalty increases, the continuity 
drops due to the longer rebuffering time caused by an increased number of buffer underflows. This shows that the 
discontinuity penalty U (t) can be used as an indicator of a lack of video continuity. 

We next present the results of the Lyapunov optimization framework described in Section IIV) labelled as LOpt 



here. We also set up our framework with a less conservative bound than the one described in ([5]), this is labelled 
as LOptM. This is done by setting = ^(*) ^rna^Pma^ -^^ ( [T4| ). We compare our framework with a typical setup 
of x264 with its target frame rate and its playout rate set to a constant 30 fps, we label this scheme Norm. We 
also compared our framework with the AMP scheme 131. We implemented a combination of AMP-Initial and 
AMP-Robust. AMP-Initial slows down the playout rate by a predetermined slowdown factor when the video starts 
playing until a certain buffer level has been reached. AMP-Robust slows down the playout rate of the video by a 
predetermined slowdown factor when the buffer level falls below a certain level. In our implementation, AMP-Initial 
is used in conjunction with AMP-Robust. We empirically chose the smallest slowdown factor of AMP such that 
it achieves a continuity of 99% or higher in each test case. This effectively simulates a close to optimal adaptive 
AMP scheme, we label this as AMP. We also added in the results of AMP with a constant slowdown factor of 25% 
(used in (Si) as a reference, which we label AMP25. 

To examine the performance of each scheme, we compare the continuity of each scheme based on the amount 
of prebuffering provided. Note that in the results, the prebuffering delay is on a log base 10 scale. Continuity is 
calculated as in ( |46l ). 

The continuity results are shown in fig. [7] It can be seen that LOpt and AMP achieves similar results. This is 
expected as AMP was tuned to achieve a high continuity. The performance disparity between AMP and AMP25 shows 
that this simulation requires a greater amount of playout slowdown at the decoder in order to reduce the occurrences 
of buffer underflows. However, both LOpt and AMP require about a 100 times less prebuffering compared to Norm 
to provide similar continuity. AMP25 too requires about 7 times less prebuffering than Norm but still requires about 
50 times more prebuffering than LOpt and AMP. This suggests that using some form of playout slowdown would 
reduce the prebuffering requirements of a video application significantly. 

We next measure the playout distortion of each scheme using ( [45] ) (p„ = 1/30), i.e. playout distortion will be 
produced when the playout interval drops below or goes above 1/30. Note that while ( [45] ) will only produce a 
non-zero value when the playout deviates from the natural playout frame interval, playout interruptions due to buffer 
underflows are not factored into the playout distortion. Fig. [8] shows the playout distortion results. Norm does not 
have any playout distortion since it has a constant 30 fps playout rate, but suffers from playout discontinuity as 
discussed earlier. LOpt has a very similiar playout distortion characteristic when compared to AMP25. In contrast, 
AMP for most cases produces twice the amount of playout distortions compared to LOpt and AMP25. This is 
mainly because AMP requires a higher slowdown factor to obtain a better video continuity and, as a consequence, 
this results in a higher playout distortion. 

LOpt's comparatively low playout distortion is due to the joint adjustment of both the encoder frame generation 
rate and playout rate. By increasing the encoder frame generation rate, the receiving frame at the decoder increases. 
This provides a higher buffer occupancy and reduces the need to slowdown the playout rate, thus reducing the 
playout distortion. This comes at the expense of frame quality, because increasing encoder frame generation rate 
will result in a higher amount of compression. To examine LOpt's effect on frame quality, we compare the PSNR of 
the encoded video before transmission. This is done to eliminate any possible drops in PSNR due to transmission. 
Norm, AMP25 and AMP have a constant encoding rate of 30 fps, this means all produce an encoded video of the 
same PSNR. Thus, we only compared LOpt with Norm. From fig. [9] it is shown that the drop in PSNR is about 
0.6 dB for LOpt. This is a reasonably small tradeoff in frame quality given the improvements in playout distortion 
and continuity. 

It can be also seen from the graphs that the performance of LOpt and LOptM are very similiar. This suggests 
that the bound in Q is not too conservative. We also tested the schemes on football, see figs. 10 12 14 and 



akiyo, see figs. [TT] [T3J [13] football and akiyo were chosen because they have the highest and lowest motion content 
respectively in the four sequences used. The results show a similiar pattern to the concatenated sequence. 

We now examine the complexity of each scheme. Norm is the least complex scheme while the AMP25 is 
marginally more complex than Norm. This is because AMP25 involves some simple logic at the decoder to slowdown 
the playout once a certain buffer level is reached. LOpt is more complex than AMP25 as it involves more calculations 
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at both the encoder and decoder end. However, it is not significantly more complex than AMP25 because the 
optimization policies are concave, so the solution search is very efficient. For example, in our implementation we 
solve for f{t) and p{t) by using the first derivatives of the encoder policy ( |20l ) and decoder policy ( |2T] ) respectively. 
AMP is the most complex in our implementation as it requires several runs to determine the most optimal slowdown 
factor. In summary, the LOpt runs in 0{n), where n is the total number of video packets. While AMP runs in 
0(n X s), where s is the number of slowdown factors to consider. However, it should be noted that complexity-wise, 
AMP is not representative of the complexity of AMP schemes in practice. Its main purpose in our simulations is 
to act as the upper bound in the performance of AMP schemes. 
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Figure 10. Prebuffering delay vs continuity (football). 
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Figure 11. Prebuffering delay vs continuity (akiyo). 
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Figure 12. Prebuffering delay vs playout distortion (football). 
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Figure 14. PSNR loss for LOpt (football). 



Figure 15. PSNR loss for LOpt (akiyo). 



C. Delay Constrained Lyapunov Optimization Results 

To evaluate the performance of our delay constrained Lyapunov optimization framework, which we call DLOpt 
here. As in the previous section, we also set up out framework with a less conservative bound ^(*) '^■r^o.^Vma^ ^ labelled 
as DLOptM. We compare our scheme with AMP-Live |3|, labelled as AMPL. AMP-Live maintains the buffer level 
by slowing down or speeding up the playout based on a predetermined scale factor. The scale factor of AMPL is set 
to 40% in our experiments, this value was found to provide the best overall performance for AMPL. The playout 
delay of each scheme is measured as: Total playout delay = X]t=o(^'(^) ~ P^)- 

Recall that p„ represents the natural playout frame interval of the sequence. So every playout slowdown will 
cause p{t) to be larger than p„, thus accumulating playout delay. To reduce the total playout delay, the scheme 
needs to find the proper moment to increase the playout rate. 

We compared the schemes by varying the delay constraints from 4.27 seconds to 273.07 seconds with the 
constraint doubled at each run. At the end of each run, we plot the total playout delay and the performance metric 
of each scheme, the playout delay in the results are on a log base 10 scale. The performance metric we examined 
are playout continuity, playout distortion and PSNR. 
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Figure 16. Playout delay vs continuity. 
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Figure 18. PSNR loss for DLOpt 
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Figure 17. Playout delay vs playout distortion. 



We first look at the continuity results with respect to the total playout delay (fig. 16 1. It can be seen that DLOpt 
achieves maximum continuity regardless of the delay constraint, while AMPL requires more than 50 times the 
amount of total playout delay compared to DLOpt to reach maximum continuity. AMPL manages the total playout 
delay constraint by maintaining a certain buffer level [3 |, thus a tighter delay constraint will result in a lower buffer 
level which has higher probabilities of buffer underflows. On the other hand, DLOpt makes a trade-off between the 
encoder frame generation rate and playout rate to satisfy the continuity goal and delay constraint. 



We next look at the distortion results in fig. 17 Again, DLOpt achieves a much lower playout distortion compared 
to AMPL. Note that the lower the total playout delay constraints the lower the playout distortion. This is because 
a low playout constraint causes DLOpt to adjust the playout rate a lot less, thus causing a low playout distortion. 
AMPL has an almost constant playout distortion. This is because, as mentioned before, AMPL only tries to maintain 
a certain buffer level and does not constrain its playout rate adjustment in anyway. 



Lastly, we examine the PSNR trade-offs made, see fig. 18 As before, we compared the PSNR of the encoded 
video of both schemes prior to transmission. It can be seen that DLOpt sacrifices about a maximum of 1 dB of 
PSNR, this PSNR drop is higher than LOpt in the previous section. The main cause of this is the additional delay 



constraint imposed using the virtual buffer p4| ). This results a lower PSNR due to the need to satisfy the delay 
constraint, however, this allows the large gains in continuity and distortion. 

Again the performances of DLOpt and DLOptM aie mostly similiar. This reinforces the possibility that the bound 
in Q is not too conservative.. 

IX. Conclusions 

In this paper, we proposed a the frame rate optimization framework. What we achieved with this framework is: 

• Performed frame rate control by a joint adjustment of the encoder frame generation interval and the playout 
frame interval. 

• Modelled the frame rate control problem using the encoder frame generation interval, the playout frame interval 
and the discontinuity penalty virtual buffer. The model is created in such a way that stabilizing the discontinuity 
penalty virtual buffer allows the video to maintain continuity. We then used Lyapunov optimization on this 
model to systematically derive the optimization policies. We showed that these policies can be decoupled into 
separate encoder and decoder optimization policies with feedback between the two. We also showed through 
experiments that the proposed discontinuity penalty based on the virtual buffer is correlated to the video 
continuity. Finally, simulation results demonstrates the effectiveness of the discontinuity penalty virtual buffer 
approach. 
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• A delay constraint imposed on the accumulated delay from playout slowdowns. We introduced the delay 

constraint into the framework using the delay accumulator virtual buffer. We showed, using Lyapunov op- 
timization analysis, that by stabilizing the delay accumulator virtual buffer, the delay constraint would be 
satisfied. Simulation results showed a superior playout continuity and playout distortion performance with a 
reasonable tradeoff in PSNR. 

• An analysis of the impact of delayed feedback from receiver to sender We derived two different analyses, 
the first analysis showed very little impact on the optimization polices. The alternate analysis showed that the 
decoder needed to use an outdated buffer state. Simulation results demonstrated that using the first analysis 
results in a better performance. 
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Appendix 

A. Derivation of Discontinuity Penalty Lyapunov Drift Bound 

We square the buffer dynamics equation ([14]), divide it by two and rearrange its terms to get: 



U{t + 1) < [U{t) - Pit) - /3(0]+ + Fifit)) 

u{t + If < u{tf + (pit) + mf + nf{t)? 

- 2U{t) \p{t) + m - F{fm (47) 
Dividing the above equation by two and rearranging, we get: 



Using the definition of the Lyapunov function ( [15] ) in the above equation yields: 

{p{t) + P{t))' + Fif{t)f 



L{U{t + l)- L{U{t)) < 
Recall that f3{t) is defined as: 



2 

U{t)[p{t) + P{t)-F{fm (49) 



Pit) = ^n^inPmin ^^^^ 

where b{t) is the amount of buffered video frames in the buffer at time t and T is the length of the whole 
sequence in time slots. Since a time slot corresponds to a frame in our framework, it can be seen that b{t) < T. 
This is because the maximum amount of video content that can exist in the buffer is the whole sequence. This then 
implies that: 

< ^^^^^P"^^" (51) 



Using the above bound in ( 149] ), we obtain: 

L{U{t + l)-LiU{t))<^ 



'^minVmin \ jp/ £/,\\2 



p(t) + ^ ) +F{f{t)) 



-U{t)\p{t) + m-F{f{t))] (52) 
Using r{t) = F{f{t)) and r{t) < Vmax, we get F{f{t)) < Vmax- Since p{t) < pmax, using these two bounds 



into the above equation yields: 

L{U{t + l)-L{U{t))<\ 



F TjYiin Pmin \ . 2 
Pmax H X — t / 



-u{t)[p{t)+m-F{fm 

= B-U{t)[p{t) + (3{t)-F{fm (53) 

where: 

1 ( 2 , ( , frain Pmin 



B=-Ar^max+{Pmax+ ) ) (54) 



If we take conditional expectations of (53 1 with respect to U{t) and use the definition of the one-step conditional 
Lyapunov drift ( [T6] ), we get: 

A([/(t)) <B- Umim +P{t) - F{f{t))\U{t)} (55) 



B. Derivation of Delay Constrained Lyapunov Drift Bound 

We first define a Lyapunov function L{X{t)) based on X{t) as: 

L{X{t)) 4 ^ 
Recall for the discontinuity penalty U{t), its Lyapunov function is: 



Then, it can be seen that ([25]) can be defined as: 

U^{t)+X'^{t) 



L{U{t),X{t)) 



2 2 

L{U{t)) + L{X{t)) 



Now, we square the buffer dynamics equation (24i to obtain 



X{t + l) < [X{t) - - td]+ + p{t) 

X{t + if < X{tf + p{tf + (pn + tdf - 2X{t) K + td- p{t)] 
Dividing the above equation by two and rearranging yields: 

7, ^ < ^ X{t) [pn + td- p{t)\ 



Using p{t) < Pmax and (56 1 in the above equation yields: 



L{X{t + 1)) - L{X{t)) < ^ (p{tf + {pn + taf) - X{t) [p„ + td- p{t)] 

= C -X{t)[pn + td-p{t)] 

where: 

= ^ (pmax + (Pn + tdf 

By adding ( [6T] ) and (53 1 together, we get: 



L{X{t + 1)) - L{X{t)) + L{U{t + 1) - L{U{t)) < 

c - x{t) \pn+td- p{t)] +B- u{t) [p{t) + m - F{fm 



Using (58 1 and rearranging the above yields: 



L{Uit + l),X{t + 1)) - L{U{t),X{t)) <B + C 

-u{t)[p{t)+m-nfm 

- X{t)[pn + td-p{t)\ 

Taking conditional expectations of the above with respects to U{t) and X{t), and using ([26]) yields 



A(C/(t),X(t)) <B + C 

-U{t)E{p{t)-F{fmU{t),X{t)} 
-X{t)E{pn + td-pit)\U{t),X{t)} 
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C. Derivation of Network Delayed Lyapunov Drift Bound 

Note that, in this section, we use the same shortened notations defined in Section [Vlj additionally we also define 
Udf = U{t + df + 1). This means that (32l can be rewritten as: 



Ud, < [U - jd,]^ + Fd, 
By squaring the above equation and dividing it by two, we obtain: 



Ul. ll + Fl 



< — + 
2-2 



Fdf 



Using the definition of the Lyapunov function ( [15] ) on the above equation and rearranging: 



L{Ud,)-L{U) 



< 



ubd,-Fd;_ 



(66) 



(67) 



(68) 



Using "f{t) = p{t) + F{f{t)) < Tma^ p{t) < Pmax and (5]_l on the above equation yields: 

Trr. 



L{Ud,)-L{U) < ^(^rL.+ I^P. 

= B'-U bds - Fd,] 



' mm Pmm 



T-t 



where: 



+ 



Tr, 



mm ymm 

T-t 



Taking conditional expectations of ( [69| ) with respects to U and using pB] ) yields: 

A{U) < B' -E{U-fd,\U} +E{UFd,\U} 



(69) 



(70) 



(71) 



D. Discontinuity Penalty Optimization Stability and Performance Bounds 

In this section, we prove that the derived encoder and decoder policies, ( |20l ) and pT] ) respectively, stabilize the 
discontinuity penalty virtual buffer. This is given by equation ( [72] ) in Theorem [T] We also show the performance 
bound of the Lyapunov optimization. 

Theorem 1. Let the long term receiving frame interval f = limj_!.oo \ X]r=o ^ {^(Ol- V network delay variation 
r{t) is i.i.(^over the timeslots, the receiving frame interval is lower bounded by the encoder frame generation 
interval as r{t) > f{t), the frame quality function is bounded as g{f{t)) < Gmax <^nd the playout distortion 
function is bounded as h{p{t)) > Hmin- Then implementing the optimization policies (20 1 and ( |21| in each timeslot 
stabilizes the discontinuity penalty U{t) (using the stability definition E{C/} = limsupj_^oo | Sr=o ^ 
and satisfies the following performance bounds: 



A/-1 



Hr, 



r=0 



liminf5(/) > {g*-h*) + Hr, 

A/— >CXD 



B 
V 



limsup/i(p) < — 



h* ) - Or. 



(72) 
(73) 
(74) 



^See Section |a] 

''with i.i.d processes, the steady state is exactly achieve every timeslot. This allows us to use Lyapunov drift analysis on a per time slot 
basis. However, Neely has shown that i.i.d processes provides all of the intuition needed to treat general processes. For more details on this, 
see chapter 4 of |[36l . 
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where V is some positive constant (i.e. V > Q), g* and h* are the specific values of g{.) and h{.) respectively 
that maximizes the objective ([8]) subjected to the constraints ([9]), ( |10[ ) and ( |1 B is defined as in ( |18[ ) and: 

AI-l 

T = 

M-1 

T = 

Proof: We first introduce A as a set of receiving frame intervals that stabilizes the discontinuity penalty U{t). 
A is bounded by rmax- Using A assumes a complete knowledge of future, but this is only required for the analysis 



on the performance bounds. With A, the optimization problem in Section IV-D can be restated as: 



Maximize: g{fit)) - h{p{t)) (77) 
Subject to: F{f{t)) = r{t) (78) 

1 

r= lim - VE{r(t)} (79) 

T=0 

re A (80) 

fmin ^ /(^) — fmax (81) 
Pmin < P{t) < (82) 

From the above optimization problem, A can be intuitively seen as a range of values that r{t) can take that 
allows the existence of at least one stationary randomized policy that can stabilize U{t). Constraint ( fTS] ) is due to 
r{t) = F{f{t)). Let /* and p* form the solution that maximizes the objective ([77]). This means that an optimal 
stable policy would ensure that the following is met: 

F(f*) = r<p* (83) 
F{f*) = f comes from constraint ( [78] ), while the inequality f < is the result of lemma 7 in |'35'|. Suppose 



now an e-optimal stable policy causes the long term receiving frame interval r to become such that: 

F{f:) = < p* - e 

fe + e G A (84) 

where e > is a positive constant and the receiving frame intervals set A^ C A. A^ can be seen as A that is 
reduced by e. Then, the e-optimal policy can be seen as optimizing the following problem: 

Maximize: g{f{t)) - h{p{t)) 
Subject to: F{f{t)) = r{t) 
r{t) G A, 

fmin ^ /(^) — fmax 

Pmin < P{t) < Pmax (85) 
Note that the long term encoder frame generation interval /* forms part of the solution that maximizes the above 



problem. Equations([83|) and ( [84| ) together implies that: 

f, < f (86) 
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This is because the e-optimal policy maximizes the same objective as ( fTTj ) but with its long term receiving frame 
interval Umited by e. Using r{t) = F{f{t)) would mean the above equation becomes: 

F{f:) < Fin (87) 

Using (|6]l and assuming there exist the long term frame interval scaling factors and e*, such that F{f*) = 
e^/* = fe and F{f*) = e*f* = f. Then, applying these properties to the above equation yields: 

eef: < e*r (88) 

Since we made the assumption that r(t) > f{t), which can be rewritten as e{t)f{t) > f{t). This implies that 
> 1 and e* > 1, which means that : 



(89) 



This is because /* is chosen by the e-optimal policy to maximize the objective in ( |85| ). Since the frame quality 
function g{f{t)) in the objective is an increasing function of f{t), this implies that the e-optimal policy will choose 
/* as large as possible. However, as /* is upper bounded by f^, which is in turn limited by e, this means that /* 
will be at most as large as /*. Furthermore, /* — /* as e — 0. This is because |[35l : 



r > 1 



r + — /: > /; 



(90) 
probabiUty 



The middle term of ( |90l ) can be seen as an example of a mixed policy that picks /* with 1 — 
and /* with probability. 

If the long term receiving frame interval is f^, then there exists a stationary randomized policy that stabilizes 
U{t) by setting the long term playout interval to p* ll36l . That is with as the long term average arrival rate into 
the discontinuity penalty virtual buffer U{t). If the policy chooses the playout interval p{t) over time such that the 
long term receiving frame interval is p* , then U{t) can stabilize since p* > r^. Thus, it will not grow infinitely 
large over time. Under such a policy, the Lyapunov one step drift bound ([T9]l would be calculated using ([84]) as: 



A([/(t)) 
< B 



< B 



V¥.{g{f{t))-h{pmU{t)} 

nu{t)m\u{t)} 
vg{f:) + um 

U{t)p* + Vh{p*) 

E{u{t)mm} 

.Vg{f:) + U{tW-e) 
U{t)p* + Vh{p*) 



(91) 



By rearranging the terms: 



AiUit))-VE{g{fit))-hipmU{t)} 

<B-Vig{f:)-h{p*))-eU{t) (92) 
<B-V{g{f:)-h*)-eU{t) (93) 

By taking expectations, summing over the timeslots r G [0 .. M — 1] and using the non-negativity of L{U{t)) to 
drop the term E{L(C/(M - 1))}, we get: 

M-l 

-E{L(C/(0))} -VY, naUir)) - h{p{T))] 



r=0 



M-l 



<MB-eY, E{U{t)] - VM{g{f:) - h*) 



(94) 



r=0 
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To prove the discontinuity penalty bound (17), we divide (|94]) by Me and rearrange its terms to obtain 



M-l „ , _ „ 



' t Emm <-'-^ 



T=0 



nL{u{o))} ^^^^ 



Me 

Taking the limits of ( [95] ) as M — )• oo and setting e = rmax yields (J2\ . Setting e = rmax is done to minimize 
the bound, as a particular choice for e will only affect the bound calculation and will not affect the policies in any 
way. 

To prove the utility bounds, note that the concavity of the frame quality function g{f{t)) and the convexity of 
the playout distortion function h{p{t)) together with Jensen's inequality implies the following: 

M-l / , M-l 

M 



M 

T=0 ^ r=0 

If we divide ([941) by MV and rearrange it, we obtain: 



- E{5(/(r))} < 4 ^ E (96) 

T=0 ^ r=0 ^ 

A/-1 . M-\ . 

- E{Mp(r))} > M ^ E ^M^)} (97) 

— n \ — n ' 



M-l , M-l 



T = T = 



>W/.-)-'.')-^-5f^2M (98) 
To obtain the frame quality bound, we use the fact that h{p{t)) > Hmin in ( |98] ) and rearrange to get: 

M-l 

E ^{^^(/(-))} 



r=0 



> [9{je ) - h )+Hmin - y (99) 



Using (96 1 and taking the limits of &% as M — oo yields 



liminf > {g{f:) - h*) + H^i^ - § (100) 
M— >-oo V 

Again, the above bound can be maximized by taking the limit as e — 0. This produces the frame quality bound 



(73l 



To obtain the playout distortion bound, we use the fact that g{x{t)) < Gmax in (98 1 and rearrange to get: 



M-l 

- 53 E{ft(p(T))) (101) 



Using (97 1 and taking the limits of ( 101| ) as M — )• cxd yields 



limsup h{p) <y- {gift) -h*)- Gmax (103) 

M-s>oo V 



Taking the limit as e — )• maximizes the above bound and produces the frame quality bound (74 1 
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E. Delay Constrained Optimization Stability and Performance Bounds 



To obtain the performance bound for the delay constrained Lyapunov optimization policies in ( [28| ), we develop 
a policy to enforce a limit on the maximum operating size of the delay accumulator. To determine such a policy, 
we observe that for the decoder objective in ([28]) (last term) to be non-negative, we need: 



Uit)p{t) 



Therefore, we introduce the following policy: 



Vh{p{t)) 
■ U{t)p{t) 



X{t)p{t) > 
X{t)p{t) > 
- X{t) < U{t) 



(104) 
(105) 
(106) 



If X{t) < u 



solve p{t) by maximizing the last term of ( |28 1 



else 

p{t) =Pn + td (107) 

Where U is a positive constant that represents how easily the maximum operating size of X{t) gets enforced. 



U is set to 100 in our experiments. (107 1 together with ( [24] ) ensures that X{t) does not accumulate anymore in 
subsequent timeslots. With ( |107| ), we introduce the following corollary: 

Corollary 2. If the network delay variation r{t) is i.i.d over the timeslots, the frame quality function is bounded 
as g{f{t)) < Gmax cifid the playout distortion function is bounded as h{p{t)) > Hmin- Then implementing the 
optimization policies from ( [28] ) in each timeslot stabilizes the discontinuity penalty U{t) and satisfies the following 
performance bounds: 



1 



A/-1 



M— >oo 



T=0 



B + C + V{Gmax ~ Hmin) 
Pmax 



liminf5r(/) > {g* 



h*) + H„ 



B + C 
V 



B -\- C 

limsup/i(p) < — — {g* - h*) - Gr, 

Af-5-oo 1/ 



(108) 
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Where V > 0, g*, r*, f and p are defined as in theorem [i] B is defined as in ( |18| ) and C is defined using C as: 



C = C + Pmax{U + Pv 



(111) 



Furthermore, implementing limit enforcing policy ( |107| ) on the delay accumulator X{t) would result in it 
deterministically upper bounded for all timeslots t as: 



X{t) <U + p„ 



(112) 



Proof: Bound ( |112| ) is proved by induction. It can be easily seen that ( |1 12| ) is satisfied at time 0. Assume that 
( 1 12[ ) holds at the current time t > 0, then we need to prove that X{t + 1) < ]l + Pmax in the next timeslot t + 1. 
We have two cases: 



1. X{t) < U : Here X{t + 1) < ]J_ + Pmax- Since from ([24]), the maximum delay added to X{t) in one timeslot 

IS Pmax- 

2. X{t) > \J_ : In this case, the limiting policy ( |107| ) will be triggered and X{t) will not increase in time t + 1 
resulting in X{t + 1) <X{t) <]l + Pmax- 

This proves ( |112| ). To prove ( |108| ), ( |109| ) and ( |110| ), observe that using ( |112| ): 
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X{t)E{pn + td-p{t)\U{t),X{t)} 
>-X{t)E{p{t)\U{t),Xit)} 



>-(U + 

Pmax n 



(113) 



Using ( [TT3] ) and ( [TTT] ) in (|28) will yield: 



AiU,X)-VE{gif)-h{p)\U,X} 
<B + C 

-¥.{Vg{f)-UF{f)\U,X} 
-¥.{Up- Vh{p) -Xp\U,X] 

The proof then proceeds exactly as in theorem [T] 



(114) 



F. Stability and Performance Bounds of Policies with Network Delays 

Corollary 3. If the network delay variation r{t) is i.i.d over the timeslots, the frame quality function is bounded 
as g{f{t)) < Gmax cind the playout distortion function is bounded as h[p{t)) > Hmin- Then implementing the 



optimization policies from ( [40] ) in each timeslot stabilizes the discontinuity penalty U{t) and satisfies the following 
performance bounds: 
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Where V > 0, g*, r*, f and p are defined as in theorem [7] B'" is defined as: 

B'" = B' + db{df + l){ 

^max ~^ TmaxPmax) 



(115) 
(116) 
(117) 

(118) 



with B' from ( 35 1 



Proof: We first look at (38). Note that with the recursive definition of U{t) (pTj), (38 1 can be rewritten as: 



A(U) < B" - U¥. {id,\U} - E {-Ud,Fd,\U} 

< B"-¥.{ (7d, [Ud, - 7]+ + F7rf,) \U] - E{-Ud,Fa,\U] 

Since [[/^^ — 7]^ < Ud^, the above equation becomes: 

A(C/) < - E { {^d,Ud, + Fjd,) \U} - E{-Ud,Fdp} 
Note that the term F^^^ can be upper bounded as: 

f^maxPmax 

Using ( [T2T] ) in ([120]) yields: 

A(C/) < B'" - E {7d,C/d, - Ud,Fd,\U} 



(119) 



(120) 



(121) 



(122) 



with B defined in ( 1 18 1. This then implies that (p9b can be bounded as: 
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AiU)-VE{g{fit))-h{pmU} 

< B" -E{U-fd,-Ud,Fd,\U} 

-E{Vg{f{t))-Vh{p{t))\U} 

< B'" -E{Ud,7df-Ud,Fd,\U} 

-E{Vg{f{t))-Vh{p{t))\U} (123) 



The proof then proceeds as in theorem [T] 



